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ABSTRACT 



This paper presents state-of-the-art models addressing 
issues related to coordination of treatment and evaluation activities, and 
integration of clinical, performance, and evaluation information. 

Specifically it provides a framework for evaluation activities to be 
integrated within planning, management, operation, and substance abuse 
treatment service delivery activities. The paper is divided into four 
sections. Part 1 provides a context for the placement of the self-adjusting 
treatment evaluation model (SATEM) concept within the evaluation framework 
described in the Integrated Evaluation Methods Package. Part 2 includes 
background information on the current role of evaluation in clinical practice 
and the potential benefits of strengthening that role. Part 3 describes 
SATEM. Part 4 describes the SATEM in relation to the Center for Substance 
Abuse Treatment's evaluation model. Appendix A, "Integrated Evaluation 
Methods Package: A Guide for Substance Abuse Treatment Knowledge -Generating 
Activities- -Executive Summary," contains a summary description of the 
Integrated Evaluation Methods Package, of which this paper is one component. 
Appendix B is "Editor's Note." (Contains 5 figure, 3 tables, and 12 
references . ) (MKA) 
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Foreword 



Over the last 10 years the Center for Substance Abuse Treatment (CSAT) has 
accumulated a great deal of experience in substance abuse treatment evaluation implemented 
through coordinating centers, cross-site efforts, and national studies. The importance and value 
of integrating ongoing evaluation activity into a system for treating substance abuse problems is 
widely recognized. Also widely recognized, however, is that current evaluation generated 
knowledge and practice are often under-utilized, due in part to the lack of an integrated approach 
to capturing information with which to measure and improve treatment effectiveness, efficiency, 
and performance. CSAT recognizes that such an integrated evaluation approach will more 
effectively support current and future knowledge generating activities. 

Based on a decade of evaluation experience, CSAT has developed the Integrated 
Evaluation Methods (IEM) Package, a series of conceptual and methodological applications, 
including concept papers, technical assistance materials, and analytic tools, to enhance CSAT- 
funded evaluation activities. Products in the IEM Package are organized within an evaluation 
framework constructed on the basis of accumulated experiences among internationally known 
treatment service evaluation professionals. Thus, the framework is based upon evaluation 
strategies, structures and approaches appropriate for substance abuse treatment evaluators and 
providers. The framework follows a standard set of evaluation activities: planning, selecting a 
design, developing data requirements and collection instruments, collecting and analyzing the 
data, and reporting the evaluation findings. (A summary description of the IEM Package is 
contained in Appendix A to this document.) 

This concept paper and its companion documents, Integrated Evaluation Methods: A 
Guide for Substance Abuse Treatment Knowledge Generating Activities; Building Team 
Capability to Fully Implement and Utilize the Self-Adjusting Treatment Evaluation Model; 
Adding “ Value ” to CSAT Demonstrations ; Performance Measurement for Substance Abuse 
Treatment Services ; and Client Levels of Functioning as a Component of Substance Abuse 
Treatment Services Evaluation, present state-of-the-art conceptual models addressing issues 
related to coordination of treatment and evaluation activities, and integration of clinical, 
performance and evaluation information. Specifically, this concept paper provides a framework 
for evaluation activities to be integrated within planning, management, operation and substance 
abuse treatment service delivery activities. 

Sharon Bishop 
Project Director 
NEDTAC 
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I. Introduction 



From the inception of the Office for Treatment Improvement, CSAT’s predecessor 
agency, Dr. Jerome Jaffe and Dr. Herman Diesenhaus recognized the need for an evaluation 
strategy that could by used to incorporate current knowledge into substance abuse treatment 
programs, the need for high quality evaluation directed to treatment improvement is strongly 
promoted by the two seminal Institute of Medicine (1990) publications on alcohol and drug 
treatment systems. 

CSAT program managers as well as evaluators and service providers from across the 
country continually reinforce what many professionals in the substance abuse treatment 
community have long stated: 

■ The quality and quantity of evaluation information need to be improved 

■ Training and technical assistance on evaluation are needed to support service 
providers 

■ A shared sense of direction and evaluation priorities should be established between 
CSAT and the treatment service providers 

■ CSAT should provide leadership in the substance abuse treatment field through 
promulgating state-of-the-art evaluation approaches and tools. 

To address these needs, this paper describes CSAT’s Self-Adjusting Treatment 
Evaluation Model (SATEM). the paper is designed to promote understanding of the Self- 
Adjusting Treatment Evaluation Model as well as to demonstrate its application within the 
treatment system. 

The Center for Substance Abuse Treatment (CSAT) supports the integration of ongoing 
evaluation within substance abuse treatment activities so as to demonstrate treatment service 
effectiveness and to improve treatment services and their outcomes. To this end, CSAT 
recommends the use of state-of-the-are evaluation methods and tools in planning, designing, and 
implementing treatment services evaluations, this document provides a description of CSAT’s 
Self-Adjusting Treatment Evaluation Model (SATEM), an approach for integrating evaluation 
findings within treatment operations so as to adjust and improve service delivery. 
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Introduction 



1 . CONTEXT FOR THE SELF-ADJUSTING TREATMENT EVALUATION 

MODEL DOCUMENT 



CSAT’s major evaluation goals are to: (1) increase knowledge about substance abuse 
treatment services; (2) improve treatment services by applying knowledge gained through 
knowledge development and application (KD&A) activities; (3) develop analytic methods and 
approaches for use in knowledge-generating activities; and (4) develop substance abuse treatment 
analysis databases. To meet these goals, CSAT has been sponsoring KD&A initiatives including 
activities that focus on homelessness, marijuana use and treatment, managed care, women and 
violence, and opioid treatment, as well as the replicability of exemplary treatment approaches 
(e.g., methamphetamine treatment) and the evaluation of best practices for targeted populations 
(e.g., exemplary adolescent treatment). 

CSAT’s evaluation experiences have reinforced the fact that substance abuse treatment 
evaluation involves a standard set of tasks that generally occur in the following order: 



■ Planning the evaluation, which includes setting the evaluation goals and objectives 
that determine the overall parameters of the evaluation 

■ Selecting the evaluation design, which sets forth the overall strategy for establishing 
the evaluation questions, measurement approach, and generalizability of findings 

■ Developing the data requirements, which flow from the evaluation questions and 
measures and include SDU, clinician, cost, and client data 

■ Developing data collection instruments, which are based on the data requirements 
and are developed or selected from a standard inventory of instrumentation 

■ Collecting the data, which includes the development of data management processes 
and tools including quality control procedures, and collecting the data 

■ Analyzing the data, which involves developing an analysis plan and conducting 
multiple levels of comparison; the analysis process is governed by the analysis plan 
and intended products and target audience(s) 

■ Reporting the evaluation findings, which includes evaluation knowledge 
dissemination and application within field. 

CSAT has directed the development of evaluation concepts, methods, and tools to support these 
evaluation tasks. The evaluation tasks and corresponding evaluation methods are summarized in 
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Appendix A, which contains a description of the Integrated Evaluation Methods (IEM) Package. 
A full discussion of the CSAT evaluation analytic framework and the other evaluation concepts 
and tools is presented in the concept paper: Integrated Evaluation Methods: A Guide for 
Substance Abuse Treatment Knowledge Generating Activities. This document and other 
products comprising the Integrated Evaluation Methods (IEM) Package are being made available 
at the Caliber Associates NEDS Web site at http://neds.calib.com. 

2. HOW THIS PAPER IS ORGANIZED 

This paper is organized into four sections: 

■ Part I provides a context for the placement of the SATEM concept within the 
evaluation framework described in the Integrated Evaluation Methods Package. 

■ Part II includes background information on the current role of evaluation in clinical 
practice and the potential benefits of strengthening that role. 

■ Part III describes the SATEM in terms of its philosophy and objectives, applicable 
evaluation designs and recommended process for design selection, data collection and 
analysis procedures, and approaches for communicating evaluation results to help 
improve or “self-adjust” service delivery. 

■ Part IV describes the SATEM in relation to CSAT’s treatment evaluation model, 
discusses the Federal role, and describes evaluation products. 

The appendices to this paper contain a summary description of the Integrated Evaluation 
Methods Package, of which this paper is one component, and an Editor’s Note. 
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II. Conceptual Framework 



While there is general consensus within the substance abuse treatment field about the 
importance of documenting treatment outcomes, traditional evaluation approaches have not 
proven effective due to the complexity of treatment issues, time constraints, and resource 
limitations. In addition, much of the information that has been generated from evaluation of 
treatment services has not been incorporated into substance abuse treatment services 
management, operations, and delivery, or shared with the field. 

1 . SUBSTANCE ABUSE TREATMENT EVALUATION 

The substance abuse treatment community generally agrees on the evaluation questions 
of greatest concern: 

■ Which treatment approaches are best? 

■ For whom? 

■ Under what conditions? 

■ At what cost? 

These questions, however, too often remain unanswered for lack of sufficient evaluation activity. 
Substance abuse treatment systems often do not fully employ evaluation for several reasons. 
Some service providers fear that evaluation may result in conclusions that are critical of their 
service delivery. Others believe that resources devoted to evaluation irresponsibly decrease the 
financial resources available for essential treatment services. Still others believe that evaluation 
requires randomized clinical trials and potentially unethical denial of needed treatment 
(Diesenhaus, 1993). 

Yet, incorporating effective evaluation into substance abuse treatment systems can 
enhance service delivery by allowing service providers to: 

■ Describe service delivery success and thereby help secure continued funding — or aid 
in attracting additional funding sources — by providing compelling evidence of the 
efficacy and efficiency of treatment services 

■ Improve day-to-day treatment operations and services and monitor the provision of 
treatment for appropriateness and cost-effectiveness 
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Conceptual Framework 



■ Objectively match clients to the appropriate treatment strategy and assess the impact 
of treatment interventions 

■ Contribute to the development of new knowledge on underlying causes of effective 
treatment for substance abuse and addiction. 



In fact, it is possible to conduct evaluation activities in a 
non-threatening ethical manner, at reasonable costs, and 
still yield highly useful information. The Self- 
Adjusting Treatment Evaluation Model (SATEM) 
offers a process through which continuous knowledge 
development and continuous improvement can take 
place. The model, as illustrated below, incorporates 
state-of-the-art evaluation techniques that emphasize 
maintaining a collaborative, continuous learning process, with broad dissemination and 
utilization of information gained to advance the substance abuse treatment field. 



The Self-Adjusting Treatment 
Evaluation Model offers a way 
in which non-threatening 
comprehensive evaluation 
can occur ethically and at 
reasonable cost. 






Self-Adjusting Treatment Evaluation Model 




DISSEMINATION and 
UTILIZATION 



Development 



EVALUATION 

ACTIVITY 
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Exhibit II- 1, on the following page, depicts the role of evaluation in substance abuse 
treatment service planning, management, and operations as they are conceptualized in the 
SATEM. 

2. EVALUATION PHILOSOPHY AND OBJECTIVES 

Evaluation is conducted on all types and all aspects of human service delivery systems. 
An evaluation may address a broad variety of important questions, including: 

■ What is the intervention or set of interventions? 

■ What is each intervention intended to accomplish? 

■ How are the interventions supposed to accomplish their objectives? 

■ Are the interventions being delivered appropriately? 

■ Are the interventions being delivered efficiently? 

■ What are the multiple impacts of the interventions, and their interrelationships, being 
delivered? 

■ How effective are the interventions? 

■ How can the interventions be improved? 

■ What treatment and services should be provided to whom, given consideration of 
costs, resources, and what yields the best outcomes for different groups of clients? 

■ What resources are required to provide optimal treatment and service to all clients? 

In answering these and other questions, however, the textbook evaluation approach has long 
conflicted with substance abuse treatment practice. The array of interventions provided by a 
treatment facility often have interrelated effects on clients and/or communities, yet the 
complexity of the interrelated effects is often lost through traditional evaluation designs and 
practices. To date, most evaluation studies greatly simplify the real world complexity of 
treatment interventions by focusing on individual interventions, or by considering interventions 
one-by-one, as if the effects of each intervention occurred in relative isolation. 
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Conceptual Framework 



In addition, evaluators have traditionally assumed the role of an “impartial observer” in 
order to maximize their own level of objectivity. In fact, recent opinion within the evaluation 
community suggests that the extent to which evaluator objectivity is actually possible has been 
exaggerated. Further, self-distancing makes it difficult — some say impossible — for evaluators to 
capture all of the important activities that affect outcomes that occur at a treatment facility. 
Intentionally placing distance between evaluators and service providers works against 
establishing a communications loop in which they help each other. Evaluators help service 
providers incorporate evaluation findings into treatment provision, and service providers help 
evaluators ensure their understanding of treatment objectives and processes, and the 
interrelationships among various groups within the treatment system. 

The traditional “impartial observer” role may be appropriate for evaluations in which 
relationships between actions and results are relatively predictable, well defined, and buffered 
from outside influences. When applied to substance abuse treatment evaluation, this approach 
leads to viewing treatment as the “black box” and does not identify the various components of 
the treatment process and their linkage to client outcomes. Because this approach does not 
adequately address substance abuse treatment and the entire substance abuse treatment system, 
the recommended evaluation approach described below, the Self-Adjusting Treatment Evaluation 
Model, is quite different from traditional evaluation. 

The Self-Adjusting Treatment Evaluation Model calls for evaluators to be considered as 
members of the treatment team, rather than “outsiders.” Evaluators who are readily accessible 
and viewed as “insiders” more easily facilitate the self-adjustment process. Studies show that 
organizations are most receptive to information on organizational processes that is generated by 
internal or familiar sources deemed credible and trustworthy (Rist, 1994). 

The self-adjusting approach to evaluation, sometimes known as “open-systems 
evaluation,” calls for a more active and intimate relationship among evaluators, service delivery 
staff, and other stakeholders who may have a personal, contractual, or legal relationship with 
treatment clients, including: 

■ Friends and family members 

■ Employers 

■ Welfare agencies 

■ Criminal justice agencies 
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■ Reimbursers such as insurers and government agencies 

■ Regulatory agencies and others who monitor legal or clinical standards 

■ Funding agencies. 

In addition, the public and their elected representatives constitute important stakeholders. 

A second component of the SATEM is an ongoing “learning community” of 
organizations. Most organizations emphasize maintaining the status quo, and therefore resist 
self-examination and change. In these organizations, learning typically takes place on an ad hoc 
basis, most often in response to sudden adversity. Most organizations also learn in isolation from 
other organizations. In contrast, organizations within a learning community collaboratively 
function in “a continual questioning mode,” encouraging dialogue between opposing views and 
perspectives, fostering creativity, and seeking input and feedback from the internal and external 
environment (Leeuw & Sonnichsen, 1994). Learning communities develop individual and 
collective skills and capabilities on an ongoing basis to improve the quality of organizational 
questioning, facilitate “fundamental shifts of mind,” and enhance the development and 
implementation of strategies that respond to what is learned (Senge et al., 1994). 

For ongoing learning to occur, an organization must have the capacity to: 



■ Sense, monitor, and scan significant 
aspects of its internal and external 
environments 

■ Relate this information to operating 
norms and values 

■ Detect significant deviation from the 
norms and values 

■ Initiate corrective action. 

The organization must also have the capacity and 
desire to routinely and rigorously examine basic 
assumptions underlying organizational policies and 
practices (Leeuw & Sonnichsen, 1994). 
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Conceptual Framework 



The SATEM calls for ongoing and reciprocal learning among the relevant stakeholders. 
This point can be illustrated by examining the evaluator’s role as an example. In the SATEM, 
the evaluator trains administrative and clinical staff in evaluation techniques and, in turn, seeks 
their assistance at all stages of the evaluation to ensure that they are selecting the most 
appropriate evaluation design, using the most efficient data collection methods, conducting the 
most important analyses, and presenting findings in the most useful manner for all stakeholders. 
Moreover, at the request of staff or other stakeholders, the evaluator offers advice and feedback 
on treatment implementation and operational issues and incorporates clinicians’ questions into 
evaluation activities. In this regard, the evaluator serves in an organizational development 
capacity. 

In addition, the evaluator should be engaged in continuous active learning and 
observation of the treatment process. Through the mechanism of process evaluation and 
treatment implementation assessment, the evaluator learns more about clinical practices and 
operational needs. This understanding, in turn, informs the evaluator’s interim conclusions and 
recommendations to improve service delivery and interpretation of outcome analyses. 



This approach helps an organization: 

■ Build evaluation capacity 

■ Integrate evaluation and treatment/ 
clinical activities 

■ Leverage evaluation resources 

■ Focus on treatment improvement 

■ Enhance the value and use of evaluation 
findings 

■ Create a more effective case 
management mechanism for assuring 
that clients are provided with 
appropriate services in a coordinated, 
effective, and efficient manner. 

The Self-Adjusting Treatment Evaluation Model 
involves developing and maintaining a bond of trust and reciprocity among all service delivery 



Evaluators have important roles 
within the Self-Adjusting Treatment 
Evaluation Model that include: 

■ Building a trusting relationship 
with service delivery staff and 
other stakeholders 

■ Monitoring data collection and 
reporting practices 



Conducting analysis 

Interpreting findings 

Incorporating stakeholder input 

Assisting in self-adjustment 
based on evaluation findings. 
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Conceptual Framework 



stakeholders. Trust and reciprocity are advantageous for all treatment purposes, but they are 
essential if treatment staff are to view evaluators as insiders. The evaluator plays a crucial role in 
establishing that bond of trust. The evaluator’s close geographical proximity to service delivery 
staff affords an opportunity, via frequent contact, to be viewed as a credible, internal source of 
information. Moreover, evaluators who cultivate good relations with staff gain an enhanced 
appreciation for the skills of clinical staff, and help build strong bridges between staff and 
evaluators at the state or national level. 

Evaluators have other important roles within the SATEM. They monitor data collection 
and reporting practices, incorporate service delivery staff and other stakeholder input into 
evaluation practice, conduct analyses, interpret evaluation findings for stakeholders, and assist in 
implementing self-adjustments based on evaluation findings. 

For a treatment system to actually be self-adjusting, service providers must maintain an 
open mind toward clinical practice and be willing to modify their practices whenever treatment 
evaluation yields findings that do not support current clinical beliefs and assumptions. Basing 
clinical practice on empirical findings helps to establish consensus among stakeholders. 

Self-adjusting evaluation requires that evaluators and clinicians keep abreast of the latest 
major findings in: 

■ Biomedical, neurobiological, and sociomedical research 

■ Psychosocial and behavioral research 

■ Clinical research 

■ Service research and evaluation 

■ Other evaluation research 

■ Policy analyses 

■ Organizational development and management practices. 

Relevant findings should be regularly incorporated into a treatment system’s clinical practice and 
evaluated in terms of their impact on treatment outcomes. 
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III. Planning and Conducting Evaluations 



Evaluation is the systematic collection, analysis, and interpretation of data about the 
implementation and effectiveness of substance abuse treatment. There are many evaluation 
design options that range from the relatively simple to the very complex. The two basic types of 
evaluation, outcome and process, use different evaluation designs. Outcome evaluations assess 
the interim and long-term treatment outcomes. Outcome evaluation seeks to assess how 
successful a treatment provider has been in meeting its goals, and it also helps to answer the 
questions about which treatment approaches are best and how effective they are for what type of 
client. 

Process evaluation describes treatment services, and monitors procedures to include 
treatment efficiency and fidelity. The process component of an evaluation calls for data to be 
collected on: 

■ Aspects of the treatment intervention(s), including theory, development, and 
application 

■ Characteristics of clients and treatment staff 

■ Resources 

■ Organizational structures 

■ Standard operating procedures 

■ Perceptions of treatment staff and participants on the effectiveness/efficiency of 
treatment services provided 

■ Internal and external factors facilitating or impeding attainment of treatment goals. 

The breadth and depth of process data to be collected is driven by the key evaluation questions 
and honed through the evaluability assessment findings and consultation with all treatment 
system stakeholders. 

Essential methods for outcome and process evaluators are summarized below. Beginning 
with the evaluation design, this section guides evaluation planning; data collection and analyses; 
data sources; data quality assurance; and communicating evaluation findings — all within the 
control of the SATEM. Additional resources that contain a detailed discussion of these 
methodological elements are listed in Exhibit I, Appendix A. 



O :SAT\CTRT END\IEM\CONCEPT\SATEM\DATA\SELFADJ WPD 

ERJC 



NEDTAC, Page 12 



20 



Planning and Conducting Evaluations 



1. DEVELOPING THE EVALUATION DESIGN 



Serving the evaluation needs of all stakeholders requires an integrated evaluation design 
that captures both process-related and outcome-related information. Even stakeholders primarily 
interested in treatment outcomes — funding agencies, for example — need process- 
related information to understand the factors which influence outcomes. An integrated 
evaluation approach allows outcome to not only be measured but also explained. When process 
analysis is used along with outcome analysis, process findings: 

■ Help clarify which components of the treatment setting would be necessary for the 
treatment effect to be observed again 

■ Reveal information about the types of 
participants who might be expected to 
exhibit the strongest or weakest 
treatment effects 

■ Suggest how the intervention should 
be modified to produce desirable 
effects more efficiently 

■ Contribute to basic theoretical 
knowledge about social behavior and social factors 

■ Suggest ways for an intervention that produces weak effects to be refined for re- 
testing 

■ Reveal unintended outcomes (Judd, 1987). 

Another important advantage of an integrated evaluation design is that it can offset the 
limitations and biases inherent in any single design and thus actually increase the chances of 
accurately describing processes and assessing impacts. 



Summative evaluations assess 
outcomes and utilize outcome 
designs. Formative evaluations 
describe programs and procedures 
and utilize process designs. CSAT’s 
SATEM calls for integration of the 
two. 



An integrated evaluation design should be developed with input from all stakeholders. 
The design should coherently tie together the many, various, and sometimes abstract steps 
involved in describing organizational and therapeutic processes and assessing the impact of those 
processes on client and community outcomes. 

Furthermore, it is through a detailed process evaluation that the evaluator and other 
members of the evaluation team gain insight into the complex interrelationships of enigmatic 



O :SAT\CTRT END\IEM\CONCEPT\SATEM\DATA\SELFADJ.WPD 



NEDTAC, Page 13 



Planning and Conducting Evaluations 



variables that affect treatment outcomes, learn the association between service delivery 
components, and can most readily explain anomalies in the treatment outcomes revealed by the 
outcome evaluation. In essence, outcomes cannot be fully explained without an understanding of 
the processes that yielded the outcomes. 

Selection of an appropriate evaluation design should be based on findings of a pre- 
evaluation analysis. That analysis helps to ensure that an evaluation will be technically feasible 
and capable of answering the evaluation questions important to decision makers by producing a 
logic model and a description of evaluation options. A logic model is a tool for understanding 
the relationship among objectives, short term outcomes, intermediate system impacts, and long 
term goals for substance abuse treatment services. The logic model is a logical series of 
statements that link conditions, the activities that will be used to address those conditions, short 
term outcomes resulting from activities, and the long term impacts that might occur as multiple 
outcomes are achieved. The logic model provides a framework for linking services to process 
and outcome goals. Understanding how the sequence of provider activities (i.e., the provider’s 
logic) and the use of resources contribute to desired outcomes forms the framework for selection 
of an appropriate evaluation design. This assessment clarifies for decision-makers the trade-offs 
between feasible approaches — such as evaluation costs versus levels of scientific rigor — so that 
the optimal choice can be made. 

2. DEVELOPING AND IMPLEMENTING AN EVALUATION PLAN 

Following selection of the evaluation design, the evaluators — in concert with key 
stakeholders — should develop a detailed evaluation plan. This “blueprint” for the evaluation 
should link the evaluation questions to data required, data sources and methods for collection and 
analysis, and identify evaluation products. The evaluation plan should describe: 

■ Theoretical framework of treatment services being evaluated 

■ The intervention(s) being evaluated (i.e., services provided) 

■ Key evaluation questions/objectives 

■ Analytic framework 

■ Specific data needed 

■ Sources of data 
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■ Data collection methods and procedures 

■ Data collection instruments 

■ Analysis methods and approach 

■ Anticipated analytic products. 

Development of the plan should include consideration of the measures that assess the extent to 
which the treatment services meet funding agency goals as well as any additional goals 
considered important by other key stakeholders. For example, one reason for a funding agency 
to administer a substance abuse treatment service might be to gain knowledge about the best 
ways to assist at-risk populations to gain access to 
substance abuse treatment. A treatment provider 
organization, therefore, would include this funding 
agency goal and assess the extent to which the 
funding agency’s goal is achieved by developing a 
set of corresponding objectives and measures. 

One such objective might be to “increase the total number of outreach contacts who enter 
treatment by 20 percent per year.” This objective would be measured by comparing the number 
of outreach contacts who entered treatment in the first year of the CSAT-funded activities and the 
number of outreach contacts who entered treatment in the year prior to the activities, and 
calculating the percentage of change. 

The following section discusses major issues surrounding the development of a detailed 
data collection and analysis plan. These issues include the types of data to be collected, both 
qualitative and quantitative, corresponding analysis methods, potential data sources, and methods 
for assuring data quality. 

3. TYPES OF DATA AND CORRESPONDING ANALYSES 



The goals of key stakeholders should 
be considered when identifying data 




The detailed evaluation plan usually specifies the types of data to be collected and the 
corresponding analyses to be conducted. Data required for evaluation usually are of two primary 
types: qualitative and quantitative. Qualitative data provide texture and content whereas 
quantitative data provide frequencies and magnitude. Examples of qualitative and quantitative 
data elements for substance abuse treatment evaluation are provided in Exhibit III- 1 . This 
section outlines data collection and analysis considerations for development of an evaluation 
plan. 
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Examples of Data Elements to Be Collected for Evaluation 
Of a Substance Abuse Treatment System 
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3.1 Qualitative Data Collection 

Rooted in anthropological research, qualitative evaluators observe people and their 
interactions with the goal of understanding how people perceive treatment services and 
why treatment services have specific effects. Qualitative evaluation is inductive, its measurement 
tends to be more subjective, and the data are case-specific and not always generalizable. When 
pooled, however, qualitative data provide a “picture” of the treatment setting, and the treatment 
process. Qualitative data also are used to assess the treatment services implementation and 
whether the services were implemented, as planned. Qualitative information is usually obtained 
through open-ended interviews, focus groups, and observation. 

3.2 Qualitative Analysis Techniques 

Part art and part science, qualitative analysis requires the careful structuring of an analytic 
framework or approach, and the development of data categories based on review of the content of 
the responses to open-ended questions. While descriptive statistics can be used when qualitative 
data are reduced, it is important not to lose sight of the content and context of the data. In 
qualitative analysis, information from one respondent may be equal in importance to the 
information from ten, if the data provided from one member offer a special perspective or insight. 
Qualitative data can provide rich texture to the quantified results. 

Qualitative analysis requires considerable judgement in determining the accuracy, validity, 
reliability, and value of the information obtained. This is assessed primarily through consistency 
and confirmation checks of one source against another and the assessed reliability of each of the 
sources. Data from secondary sources (e.g., agency statistics) are primarily useful in confirming 
statements made by primary sources (e.g., agency officials). This is often referred to as 
triangulation. 

3.3 Quantitative Data Collection 

Quantitative analysis uses methods from the physical sciences and includes statistical 
techniques to evaluate treatment service outcomes and objectives. Evaluators should include 
measures for both interim or intermediate outcomes and long-term outcomes. 

Evaluation literature defines intermediate outcomes as changes resulting from the 
intervention that occurs during treatment participation, by completion of the treatment episode 
during aftercare or shortly following treatment completion. Interim outcomes are defined as those 
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changes occurring between intake and exit or termination of a treatment episode. Interim 
outcome data should be an integral part of an ongoing evaluation feedback cycle. Periodic 
discussion between evaluators and clinicians of interim outcomes data helps to ensure that 
evaluators correctly interpret responses to treatment and provide clinicians and treatment service 
managers with a swift and empirically-based opportunity to revise or “fine tune” treatment 
practices or approaches. 

Long-term outcomes are the long-term effects of a treatment episode; the product of 
intermediate outcomes. These types of measures are of primary interest to stakeholders with a 
contractual or legal relationship with treatment clients or treatment facilities, including criminal 
justice agencies, reimburses, funding agencies, and legislators. Long-term outcome data, 
regularly disseminated to these stakeholders as part of the self-adjusting process, help ensure 
support for the treatment system. 

Both intermediate and long-term outcome measures may target individuals, organizations, 
or even entire communities, depending on the nature and objective of the intervention being 
evaluated. At the individual level, outcome measures generally focus on the set of characteristics, 
attitudes, and behaviors targeted by the intervention. The difference between interim or 
intermediate and long-term measures at the individual level is generally the number of months 
past the completion of the intervention. In fact, change at the individual level is best measured by 
collecting data on the same measures at intake, and at exit/termination for interim outcomes and at 
follow-up (either 6 or 12 months after exit/termination) for long-term outcomes. The individual 
outcome measures typically used in substance abuse treatment program evaluation include: 

■ Level and type of substance use and behaviors 

■ Physical and mental health status 

■ Criminal activity and criminal justice system status 

■ Social and family functioning 

■ Education/employment status 

■ Housing/living conditions. 



O ;SAT\CTRT END\IEM\CONCEPT\SATEM\DATA\SELFADJ.WPD 

ERIC 



27 



NEDTAC, Page 18 



Planning and Conducting Evaluations 



At the organization level, intermediate outcomes could include: 

■ Number and type of agencies with cooperative agreements for client referrals 

■ Percentage of clients reporting satisfaction with services received 

■ Services received 

■ Treatment completion rate 

■ Average length of stay 

■ Cost per client. 

Long-term outcomes could include: 

■ Rate of substance abuse recidivism 

■ Rate of criminal recidivism 

■ Number of funding sources 

■ Level of support from public agencies and private insurers 

■ Level of funding. 

The dividing line between intermediate and 
long-term outcome measures is to some 
extent arbitrary, yet the process of creating a 
division helps all stakeholders understand the 
treatment intervention logic and ensures that 
the full range of intended effects are 
considered for assessment. 

3.4 Quantitative Analysis Techniques 

Quantitative analysis techniques range from the simple to the sophisticated. For example, 
descriptive analyses of surveys, records, or interview data often merely require summing numbers 
and calculating percentages. On the other hand, explanatory analyses are frequently based on 
more complex techniques such as multivariate analyses or causal modeling. Such advanced 
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episode. 



% 

l 



Long-term outcomes are effects 
measured at follow-up (for example, 6 
12 months after treatment exit or 
termination). 



E M SBE 






o 



SAT\CTRT_END\IEM\CONCEPT\SATEM\DATA\SELFADJ.WPD 



NEDTAC, Page 19 



Planning and Conducting Evaluations 



analyses should be conducted by individuals who have both formal training in statistics as well as 
significant evaluation experience. 

Implementation of the SATEM requires that the evaluation team be familiar with 
traditional as well as state-of-the-art approaches to conducting qualitative and quantitative 
analyses to help ensure that treatment evaluations meet the highest standards and yield useful 
findings at a reasonable cost. Evaluations are more likely to be properly implemented if other 
stakeholders are also conversant in advanced analytical techniques or if these techniques are 
explained in lay terms. 

4. DATA SOURCES 

An evaluation that assesses both processes and outcomes typically requires several data 
sources; therefore, data sources must be developed or identified for each substance abuse 
treatment service component to be included in the evaluation. For example, treatment facilities 
that operate a community outreach component should identify or develop data sources for data on 
outreach activities, as well as intake, treatment exit, and continuing care/aftercare components. 
Finally, data sources for follow-up must be developed or identified for evaluations designed to 
assess long-term outcomes. 

To minimize evaluation costs, evaluators should maximize their use of pre-existing data, 
such as treatment service reports and records, including operational/clinical records. 

Nevertheless, it is often necessary to generate specific data for the evaluation to assess 
intermediate and/or long-term outcomes. 

In designing and implementing a self-adjusting treatment evaluation, evaluators should 
work with treatment staff to identify credible, non-intrusive data sources. For example, evaluators 
frequently create new data sources by working with clinical staff to revise data collection forms to 
capture process or outcome data. Examples of forms which may be revised to capture process or 
outcome data include: 

■ Intake forms 

■ Staff rating forms 

■ Client self-rating forms 
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■ Exit forms 

■ Post-treatment service tracking forms. 

Treatment staff are generally receptive to revising their forms if they understand how doing so can 
enhance monitoring treatment operations and better demonstrate the effectiveness of the treatment 
approach to potential funding sources. 

For process analysis, written documents often yield rich data on how and why a treatment 
approach was developed and implemented. Early in the evaluation process, funding applications, 
progress reports, interagency agreements or correspondence, official policies and procedures, and 
client records should be reviewed for potential use as evaluation data as well as to verify one’s 
understanding of all treatment goals. Other data sources on provider processes include central 
intake and referral records. Additional data sources on treatment objectives and processes may be 
developed by conducting surveys or interviews with treatment staff or clients or by recording 
observations of provider operations. Open-ended questions posed to staff, via personal 
interviews, are particularly useful to reveal the nuances behind operating decisions and practices. 

For analysis of interim outcomes, several pre-existing data sources may be tapped, 
including treatment records and client exit forms. New data may also be required to assess 
interim outcomes. Such data may be collected through surveys or interviews with clients as they 
leave treatment or shortly thereafter (within 1 month), or by accessing the records of other 
organizations such as welfare agencies, criminal justice agencies, hospitals, or schools. 

Similarly, for analysis of long-term outcomes, existing records from several different 
sources may be utilized. For example, treatment service records on participant behaviors may 
serve as the data source for assessing interim outcomes while state welfare system or criminal 
justice systems records may serve as the data source for assessing long-term outcomes. Data 
sources used to assess long-term outcomes should allow for matching data elements at the intake, 
treatment, treatment exit, and follow-up stages. This allows assessment of changes in client 
conditions — for example, physical health, mental health, employment status, criminal justice 
status — over time. 
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Types of Analysis and Examples of Data Sources 

Type of Analysis Data Sources 

Process Analysis Written materials often provide insight on how 

and why specific treatment interventions were 
developed and implemented. 

Personal interviews in the form of open-ended 
questions may help to reveal nuances behind 
operating decisions and treatment practices. 

Interim Outcome Analysis Treatment records and client exit forms may be 

tapped. New data can be obtained through 
client surveys or interviews (within 1 month of 
leaving treatment), or from records of hospitals, 
welfare agencies, criminal justice agents, or 
schools. 

Long-term Outcome Analysis Existing records from several sources may be 

used, including State welfare and criminal 
justice records. 

Outcome data sources should allow for 
matching data elements at intake, during 
treatment, at exit, and follow-up stages in order 
to assess changes in client conditions over 
time, e.g., physical and mental health, 
employment status, and criminal justice status. 

In addition to objective measures of interim and long-term outcomes, evaluation studies 
frequently collect follow-up data from clients through face-to-face interviews and/or telephone or 
mail surveys. Primary concerns for this follow-up data source include: 



■ Cost — follow-up data collection can be very costly depending on the ease of locating 
the clients 6 to 12 months following discharge 

■ Sample size — sample size must be sufficiently large to allow for missing data from 
hard to locate clients 
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■ Accuracy of self-report data should be increased through use of key objective 
measures (e.g., urine tests to substantiate self-report of substance use). 

All of these technical and procedural issues can be greatly alleviated through treatment staff/ 
evaluator coordination in planning and implementing follow-up data collection mechanisms early 
in evaluation planning. CSAT has produced tools and other aids as part of the Integrated 
Evaluation Methods package to help program staff and their evaluators overcome barriers to 
successful follow-up data collection. (See Appendix A.) 

5. METHODS FOR ASSURING DATA QUALITY 

Assessing the same phenomena based on more than one data source is a concept known as 
“triangulation of data sources.” Social scientists have borrowed the term “triangulation” from the 
field of navigation to help describe how the use of multiple approaches to an evaluation question 
can enable an investigator to “zero in” on the information sought. For social scientists, the logic 
of triangulation applies to situations in which two or more dissimilar evaluation designs, 
measuring instruments, or data sources are used. The key to triangulation is the use of dissimilar 
methods or measures which do not share the same methodological weaknesses. If dissimilar 
methods produce the same findings, then the confidence about the results increases. 

The SATEM helps to ensure data quality. Frequent and collaborative stakeholder 
interaction helps reinforce evaluation goals and benefits and treatment staff may be more 
motivated to collect and ensure quality data. Frequent interaction between evaluators and 
treatment staff can be facilitated via new technology. For example, an Internet, electronic bulletin 
board system or another electronic conference system would allow for swift and inexpensive 
communication between evaluation team members and treatment staff, though they might be 
physically located at different sites. The on-line system can be used to discuss problems, 
exchange ideas, review evaluation issues, and pose questions for immediate feedback. Moreover, 
an electronic management information system can be developed to allow for the ongoing 
reporting of data and an immediate error-checking capability and feedback cycle. 

6. COMMUNICATING EVALUATION FINDINGS 

The SATEM calls for frequent and widespread communication of evaluation findings so 
that stakeholders may make frequent and widespread use of the information to improve program 
operations and the delivery of services. Evaluation findings and recommendations must be easily 
understood if they are to ultimately enhance treatment operations and outcomes. Therefore, 
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evaluators must pay careful attention to both the content and format of evaluation reports and 
communications. Evaluators should regularly share evaluation products with stakeholders via the 
informal and formal mechanisms described below. 

6.1 Informal Communication of Evaluation Findings 



evaluation data suggest either a better 

approach or that the current approach is less effective than desired. Informal and frequent 
feedback on evaluation data helps assure the integrity of the evaluation and assists evaluators in 
describing the evolution of treatment systems, to provide a vivid, moving image of the treatment 
approach rather than the traditional one-moment-in-time snapshot. 

6.2 Formal Communication of Evaluation Findings 

Evaluators should provide evaluation results formally through three basic media: 

■ An Executive Summary which highlights the key findings, conclusions, and 
recommendations. These summaries are generally intended for treatment system 
leadership to facilitate decision-making, as well as for broader distribution to other 
organizations and agencies — any stakeholders who may benefit from the information. 

■ A detailed evaluation report which documents the background and objectives of the 
study, the methodology used, the findings obtained, and the conclusions and 
recommendations reached. The detailed report is intended for use by managers and 
treatment staff who will want to understand how and why conclusions were reached 
and other evaluators who may be interested in the details of an evaluation. 

■ Briefing materials which synopsize the findings, conclusions, and recommendations in 
a briefing format. The level of detail will be similar to that presented in the Executive 
Summary, but designed for oral presentation, and consisting of short, up-front 
summaries of findings, pictorial display of data, and action-oriented conclusions and 
recommendations. 

Development of executive summaries, evaluation reports, and briefing materials should be guided 
by the following general principles: 
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■ Know the audience in terms of what information is needed and why 

■ Make the presentation at the appropriate level of technicality 

■ Enhance the credibility of findings by explaining the study methods and by using 
examples. 

These practices help to ensure an effective use of evaluation. The IEM package summarized in 
Appendix A includes a compendium of outlines for evaluation products including interim 
evaluation reports, final evaluation reports, replication studies, case studies, and ethnographies. 
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IV. The Self-Adjusting Treatment Evaluation Model 
and CSAT’s Evaluation System 



One goal of the Center for Substance Abuse Treatment (CSAT) is to increase the 
knowledge about substance abuse treatment and to improve treatment and treatment systems 
through innovative demonstrations and their evaluations. Since FY 1990, CSAT has evolved both 
in the depth, breadth, and focus of its demonstrations and in providing increasingly pragmatic yet 
scientifically rigorous evaluations. As a part of the treatment demonstration and evaluation effort, 
CSAT has developed and subscribed to the concept of the Self-Adjusting Treatment Evaluation 
Model (SATEM). 

Within CSAT, the Program Evaluation Branch (PEB) serves as the focal point for 
developing and disseminating state-of-the-art treatment evaluation designs and methods. The 
SATEM is one of the principle evaluation concepts that PEB is promulgating for the evaluation of 
treatment services. 

The purpose of this section is to describe CSAT’s operationalization of the SATEM within 
an overall, integrated evaluation system. This chapter begins by providing a description of 
CSAT’s integrated evaluation system. Also included in this chapter is a description of CSAT- 
recommended evaluation products. 

1. CSAT’S EVALUATION SYSTEM 

As a Federal leader in substance abuse treatment improvement and knowledge generation, 
CSAT has responsibility for creating increasingly effective treatment services, and for evaluating 
treatment effectiveness and cost-effectiveness, thereby adding to the body of knowledge about 
substance abuse treatment. CSAT program managers are responsible for designing new treatment 
systems, implementing these systems, and generating knowledge development and application 
(KD&A) activities, via CSAT demonstration grants. CSAT policy analysts and evaluation 
specialists are responsible for assessing the treatment efficiency and cost-effectiveness of 
treatment services. These CSAT staff also have more global responsibility to facilitate the 
application of increased knowledge about substance abuse treatment and to identify and 
disseminate “best practices” to the community at large. 

To this end, CSAT’s Program Evaluation Branch (PEB) developed and implemented an 
integrated evaluation system. This integrated system, graphically depicted in Exhibit IV- 1, 
consists of the following major evaluation activities: 
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CSAT’S Integrated Evaluation System 
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■ KD&A grant monitoring support: PEB staff provide KD&A grant consultation to 
all CSAT organizational units by contributing evaluation concepts, designs, and 
protocols which are incorporated into the Guidelines for (Grant) Applications (GFAs); 
participating in local, regional, and national advisory committees for knowledge 
development and application activities; representing CSAT at national and regional 
meetings of professional associations; and providing regular, routine consultation to 
CSAT program managers to support KD&A grant monitoring. 

■ Knowledge development: PEB supports CSAT’s substance abuse treatment 
knowledge development through scientific activities, such as the National Evaluation 
Data System efforts, which acquire national, regional, and local substance abuse 
treatment evaluation data sets, construct interrelational analysis data sets, and conduct 
analyses and meta-analyses to address treatment services effectiveness and efficacy 
questions and to inform Federal and local policy decisions. 

■ Knowledge application: PEB applies knowledge gained through KD&A and other 
analysis activities (including evaluation concepts, methods, and tools) to current 
substance abuse treatment KD&As and treatment services evaluations; in addition, 
knowledge gained from evaluations about treatment services effectiveness is applied to 
new CSAT initiatives through the CSAT KD&A program development process. 

■ Methods development and testing: PEB develops and tests specific concepts, 
methods and tools which are designed to advance the state of the art in substance abuse 
treatment evaluation and to enhance the quality of evaluation practice; for example, 
development of a cost analysis methodology to capture unit costs of different treatment 
services, development of an approach to identify geographic areas of greatest need for 
substance abuse treatment services, and development of adult and adolescent intake 
assessment instruments to enhance the collection of common data elements. 

■ Treatment services evaluation: PEB sponsors a wide array of substance abuse 
treatment service evaluations including national evaluations, CSAT program 
evaluations, and special evaluations. For example, the National Treatment 
Improvement Evaluation Study (NTIES) was designed to evaluate all FY 1991 grant 
funded programs, nationwide, and three cross-site evaluations were designed to 
evaluate specific FY 1995 CSAT programs. PEB also provided evaluation technical 
assistance through the National Evaluation Data and Technical Assistance Center 
(NEDTAC) including evaluation training, evaluation clearinghouse support, and data 
collection and management support through the development and implementation of 
automated data systems for CSAT treatment service providers. 

■ Treatment improvement and outcome analysis: in addition to national, program- 
wide and special evaluations, PEB sponsors analyses of national evaluation databases 
to address specific policy issues such as substance abuse among adolescents, substance 
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use patterns among older Americans, effectiveness of specific treatment services for 
distinctive populations, and treatment services outcomes for specific substances of 
abuse. 

■ Product development: all evaluation efforts supported by PEB are summarized, 
synthesized, and packaged for potential distribution among Federal, state, and local 
policy makers, service providers, researchers and evaluators, and the general public. 
Evaluation products are made available through clearinghouses; national, regional, and 
local conferences; newsletters; and websites. 

In addition, CSAT’s Synthesis Branch uses PEB products for broader synthesis and dissemination 
to the treatment field. PEB works closely with CSAT’s Scientific Analysis Branch and 
SAMHSA’s Office of Applied Studies to ensure that a full spectrum of analyses are available to 
the substance abuse treatment community. 

2. EVALUATION PRODUCTS 

The CSAT evaluation system has adopted a broad view toward evaluation products. 
Typically, evaluation products have included annual evaluation progress reports and final 
evaluation findings, conclusions, and recommendations. Other products have included briefing 
packets for the final evaluation report and any publications resulting from the final report. 



CSAT defines evaluation products 
more broadly to include any data and/or 
documentation resulting from each phase of 
the evaluation process. This shift in the 
definition of evaluation products is primarily 
designed to facilitate the identification of 
evaluation approaches, tools, findings, and 
results as early as possible, so that program 
managers and staff can routinely integrate 
evaluation findings into treatment operation 
improvements. A secondary advantage of this 
approach to evaluation products is that CSAT 
can serve as a repository for evaluation information, methods, and results and thereby serve as a 
central resource to the substance abuse treatment field regarding state-of-the art evaluation 
activities. Currently, CSAT makes available a number of evaluation conceptual and resource 
materials to enhance evaluation product development. The materials are contained within the 
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Integrated Evaluation Methods (IEM) package, and are briefly summarized in Appendix A to this 
paper. 



A diagram of treatment phases, evaluation phases, and corresponding evaluation products 
is presented in Exhibit IV-2. As shown, every major evaluation activity results in an evaluation 
product which is useful to CSAT and program management for the self-adjusting system and for 
monitoring evaluation progress. Examples of these products include: 

■ During the services planning and evaluation planning phases, the evaluation would 
provide: 

Evaluation plans 

Data collection tools and procedures 
Analysis plans. 

■ As the evaluation is implemented and treatment services are provided, the process 
evaluation would be conducted. Products from this phase are: 

Process evaluation report (interim and final) 

Intervention manuals 

Ethnographies, case studies 

Interim evaluation datasets 

Interim statistical reports 

Progress reports, briefings, and other documents containing interim evaluation 
information, 
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Exhibit IV-2 

Evaluation Products Resulting From CSAT Self-Adjusting Treatment 
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■ As clients complete treatment or otherwise terminate treatment, the evaluation will 
capture outcome and impact data which would contribute to the following products: 

Annual and final outcome evaluation reports 

Final evaluation data sets 

Cost data analyses 

Publications 

Briefings. 

The final evaluation products would be produced at the conclusion of the evaluation study. The 
final products would include the final evaluation report and supporting documentation, final 
evaluation data sets and databases, cost analyses, publications, and final briefings. 

3. SUMMARY 

In 1992, with the establishment of the Program Evaluation Branch (PEB), an evaluation 
strategy was promulgated which pivoted on the self-adjusting treatment evaluation model. This 
strategy requires highly interactive evaluation and program management and administrative 
functions. The CSAT program area management, according to the strategy, is informed by 
progress reports and annual evaluation products and processes, including process evaluations of 
treatment implementation and interim and long-term outcome evaluations. 

PEB developed the National Evaluation Data and Technical Assistance Center (NEDTAC) 
to institutionalize an overall evaluation system with primary responsibility for operationalizing 
the self-adjusting treatment evaluation model. NEDTAC developed procedures, tools, and 
mechanisms to assist all CSAT treatment service providers with these important evaluation 
concepts, principles, and methods. To assist CSAT and its treatment service providers in 
implementing the self-adjusting treatment evaluation model, all of the information described 
above was made available through training and technical assistance services. These efforts 
produced a series of evaluation products that will last beyond NEDTAC itself. 

CSAT’s Program Evaluation Branch will continue to use all of the existing evaluation 
products, supplemented by newly created products, to extend the value of the SATEM and 
maintain its integration within the overall CSAT evaluation system. These products are described 
in the paper entitled Integrated Evaluation Methods: A Guide for Substance Abuse Treatment 
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Knowledge-Generating Activities and combined within the Integrated Evaluation Methods 
package described in Appendix A to this paper. The IEM package represents a comprehensive 
evaluation approach which is tailored to CSAT KD&A and substance abuse treatment evaluation 
perspectives. The integrated methods package begins with evaluation planning and thereby 
insures the incorporation of the self-adjusting treatment evaluation model concepts throughout the 
evaluation process. 

Building on the foundation laid by the NEDTAC evaluation technical assistance and 
support activities, PEB is supporting the national field of substance abuse treatment evaluation by 
promoting the standardization of evaluation concepts, methods, tools, data specification, and data 
collection. Through these integrated efforts, PEB is ensuring that substance abuse treatment 
evaluation activities are coordinated so as to yield comparable evaluation data and data analyses 
that will address national substance abuse treatment efficacy policy issues as well as concerns 
about unique populations, specific substances of abuse, and specialized treatment sendee 
approaches. 
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Appendix A: 

Integrated Evaluation Methods Package: 

A Guide For Substance Abuse Treatment 
Knowledge-Generating Activities — Executive Summary 

Since its inception, the Center for Substance Abuse Treatment (CSAT) has provided 
Federal leadership to improve substance abuse treatment accessibility, effectiveness, and 
efficiency. CSAT’s mission and activities have evolved from directly supporting treatment 
services to supporting knowledge-generating activities. This evolution is evident in the current 
Substance Abuse and Mental Health Services Administration policy on evaluation as described 
in Evaluation Policy, SAMHSA, 1995. 

The need for an integrated model of evaluation and planning at SAMHSA is presented in 
“Evaluation in the Substance Abuse and Mental Health Services Administration,” Evaluation 
and the Health Professions, by Marsh, Jansen, Lewis, & Straw, 1996. CSAT also supports site- 
specific, cross-site, and national evaluations that have provided experience with a wide array of 
evaluation design and implementation methods. These experiences further supported the need 
for an integrated evaluation strategy and led to the development of a comprehensive set of 
evaluation products, including concept papers, technical assistance (TA) materials, and analytic 
tools. Collectively, these products are referred to as the Integrated Evaluation Methods (IEM) 
Package. The IEM Package organizes these products within an evaluation framework that is 
designed to support CSAT knowledge development and application goals. The evaluation 
framework itself was constructed on the basis of accumulated experiences among internationally 
known treatment service evaluation professionals. The IEM Package reflects and incorporates 
evaluation experiences gained over the past decade. 

Evaluation Framework and the Integrated Evaluation Methods Package 

National evaluation experiences have reinforced the fact that substance abuse treatment 
evaluation involves a standard set of tasks that generally occur in the following order: 

■ Planning the evaluation/knowledge-generating activities, which includes selecting 
the substance abuse treatment issue, identifying the theoretical foundation for the 
intervention, determining knowledge development program goals and implementation 
approach, and setting the evaluation goals and objectives that determine the overall 
parameters of the evaluation 
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■ Selecting the evaluation design, which sets forth the overall strategy for establishing 
the process and outcome evaluation questions, measurement approach, and 
generalizability of findings 

■ Developing the data requirements, which flow from the evaluation questions and 
measures and include: SDU, clinician, cost, and client data 

■ Developing data collection instruments, which are based on the data requirements 
and are developed or selected from an integrated inventory of instrumentation 

■ Collecting the data, which includes developing data management processes and tools 
(including quality control procedures) and conducting the data collection activities 

■ Analyzing the data, which involves multiple levels of comparison and is governed 
by an analysis plan 

■ Reporting the evaluation findings, which includes evaluation knowledge 
dissemination and application within the field. 

The evaluation process outlined above provided a framework for the development of products 
related to these evaluation concepts and methods. Taken together, those products comprise the 
IEM Package. 

Integrated Evaluation Methods Products 

CSAT requested the development of a series of evaluation concept papers, TA materials, 
and tools to support and operationalize each phase in the evaluation of substance abuse treatment 
knowledge-generating activities. These items are included in the IEM Package. The concept 
papers are based on theoretical evaluation research constructs that have been adapted to 
substance abuse treatment services evaluation and knowledge-generating activities. The concept 
papers primarily support the evaluation planning phase and address such topics as the self- 
adjusting treatment evaluation model, cost analyses, and performance measurement. The TA 
materials and tools include specific evaluation methods that have direct applicability to substance 
abuse treatment knowledge-generating activities. The concept papers and TA materials that 
constitute the IEM Package are listed and briefly described in Exhibit I. 
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Exhibit I 

Evaluation Framework and Integrated 
Evaluation Methods Package 



Appendix A 



evaluation 

FRAMEWORK 


integrated EVALUATION METHODS PRODUCTS 


1. Planning the 
evaluation/ 
knowledge- 
generating 
activities 


■ Integrated Evaluation Methods: A Guide for Substance Abuse Treatment 
Knowledge Generating Activities: Concept paper that describes the development of an 
evaluation framework, evaluation concepts, and TA materials to support the framework. 

■ Self-Adjusting Treatment Evaluation Model: Concept paper that describes an 
approach for integrating evaluation findings within treatment operations so as to adjust 
and improve service delivery. 

■ Building Team Capability to Fully Implement and Utilize the Self-Adjusting 
Treatment Evaluation Model: Concept paper to assist treatment providers in building 
capabilities to integrate the self-adjusting treatment model within day-to-day operations 
and service delivery. 

■ Adding “Value” to CSAT Demonstrations: The What, How and Why of Cost 
Analysis: Concept paper on the need for and types of cost analyses for CSAT 
demonstrations and knowledge-generating activities. (The Lewin Group) 

■ Performance Measurement for Substance Abuse Treatment Services: Concept 
paper about the increasing importance of provider performance measurement and 
analyses and an explanation of the case-mix adjustment methodology. 

■ Client Levels of Functioning as a Component of Substance Abuse Treatment 
Services Evaluation: Description of the rationale and methods for assessing client level 
of functioning and recommended core LOF data elements that could help to measure the 
effectiveness of treatment services received. 

■ Substance Abuse Treatment Evaluation Policy Notebook: These materials are aimed 
at facilitating understanding of the SAMHSA policy for evaluation and federal 
regulations on client confidentiality and assisting evaluators to meet CSAT evaluation 
requirements. 

■ Substance Abuse Treatment Evaluation Resource Notebook: The notebook contains 
evaluation bibliographies and listings of organizations, hot lines, on-line data bases, and 
contact information for obtaining assistance in evaluating treatment services. 


2. Selecting the 
evaluation design 


■ A Guide to Process Evaluation for Substance Abuse Treatment Services: TA tool 
presenting purposes of process evaluation and the application of process evaluation 
methods to single site and multi-site treatment services. 

■ Using Logic Models in Substance Abuse Treatment Evaluations: TA tool describing 
logic model purposes and techniques for designing and planning the evaluation of 
treatment services. 

■ A Guide to Selecting an Outcome Evaluation Design for Substance Abuse 
Treatment Evaluations: TA tool describing overall strategies for developing 
evaluation questions, measurement, controls, validity/reliability, sampling, design 
effects, and generalizability of findings. (Battelle) 
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Exhibit I (Continued) 
Evaluation Framework and Integrated 
Evaluation Methods Package 
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EVALUATION 

FRAMEWORK 


INTEGRATED EVALUATION METHODS PACKAGE 


3. Developing data 
requirements 


■ Minimum Evaluation Data Set (MEDS): Core Data Lists: TA tool for developing a 
uniform set of variables and response categories for the service delivery unit (SDU), 
clinician, cost, and client evaluation measures. 

■ Substance Abuse Treatment Cost Allocation and Analysis Template (SATCAAT): 

User manual to analyze treatment costs by unit of service for an SDU. (Capital 
Consulting Corporation) 


4. Developing data 
collection 
instruments 


■ Substance Abuse Treatment Services Evaluation Data Collection Instruments: Data 
collection instruments that fully incorporate the MEDS and that have been field tested 
for validity and reliability, as follows: Service Delivery Unit (SDU) Description; 
Clinician Background and Practice Survey; protocols to collect Adult, Adolescent and 
Child (in treatment with parent) Client Data at Intake, During Treatment, at Treatment 
Discharge and Post Treatment; Adult and Adolescent Record Extraction forms; and a 
section on protection of human subjects and informed consent. 


5. Collecting the 
data 


■ Staying In Touch: A Fieldwork Manual of Tracking Procedures for Locating 
Substance Abusers for Follow-up Studies (UCLA): User manual to establish and 
implement client follow-up data collection systems and procedures. 

■ Strategies for Follow-up Tracking of Juvenile, Homeless, and Criminal Justice 
System-Involved Substance Abusers: Overview and Bibliographies, 1990-1998: 
Description of tracking techniques used to increase response rates for follow-up 
interviews with homeless and juvenile/criminal justice involved substance abusers. 


6. Analyzing the 
data 


■ A Guide to Substance Abuse Treatment Evaluation Data Analysis: Recommended 
methods and procedures for analyzing process, SDU, clinician, cost, and client 
evaluation data. 


7. Reporting the 
evaluation 
findings 


■ Substance Abuse Treatment Evaluation Product Outlines Notebook: Compendium 
of outlines forevaluation products including evaluation plans, interim evaluation reports, 
final evaluation reports, replication studies, case studies, and ethnographies. 
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CSAT Evaluation “Stakeholders” 

Evaluation “stakeholders” are individuals, groups, or organizations that have a significant 
interest in how well a program or activity functions. (See P.H. Rossi, H.E. Freeman, & M.W. 
Lipsey, Evaluation: A Systematic Approach, 6th Edition, 1999.) Within the context of the IEM 
Package, CSAT evaluation stakeholders include CSAT senior managers, CSAT project officers, 
and CSAT grantees and contractors including treatment service providers, coordinating centers, 
study sites, site-specific evaluators, and national evaluators. 

Utility of the IEM Package for CSAT Evaluation Stakeholders 

While the conceptual and TA materials were developed from the perspective of the site- 
specific and multi-site evaluator, the concepts and TA tools have important utility for CSAT 
managers, project officers, and treatment service providers. The stakeholder’s position 
determines the perspective and utility of the IEM Package concepts and tools. For example, a 
CSAT senior manager can use the IEM Package to acquire a comprehensive evaluation context 
for planning and funding the knowledge-generating activities, the project officer can use the IEM 
Package to ensure that GFA/RFP applications are complete and include a full complement of 
design, execution, and product components, and the site-specific and multi-site evaluators can 
use the IEM Package to ensure that evaluation designs, data collection plans, data analyses, and 
product development have a consistent evaluation framework and compatible data across 
program areas. The suggested utility of the IEM Package for CSAT evaluation stakeholders is 
summarized in Exhibit II. 
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Exhibit II 

Utility of IEM Package for CSAT Evaluation Stakeholders 



STAKEHOLDERS 


ROLES AND RESPONSIBILITIES 


IEM PACKAGE UTILITY 


SENIOR 

MANAGERS 


■ Policy development 

■ Issue identification for KD&As 

■ Grant/contract funding decisions 

■ Overall program management 

■ Sustainability 

■ Dissemination 

■ Long-term strategic planning 

■ Program designs 

■ KA activities 


■ Comprehensive evaluation framework 

■ Comprehensive evaluation components 

■ Roles and responsibilities for local/national 
evaluators as well as CSAT/grantee staffs 

■ Guidance for evaluation designs and 
products 

■ Standardized evaluation measures 

■ Logic models for program and evaluation 
design 


PROJECT 

OFFICERS 


■ GFA/SOW development 

■ Grant/contract application review 

■ Grant/contract monitoring 

■ Kmowledge-generating products 

■ Identification and replication of 
promising practices 

■ Technical assistance assessment 


■ Guidelines for high-quality evaluation 
designs (process and outcome) 

■ Logic models for program and evaluation 
designs 

■ List of evaluation measures with 
instrumentation 

■ Guidelines for evaluation products 


GRANTEES: 
STUDY SITES 


■ Grant applications 

■ Project development, implementation 

■ Local evaluation management 

■ Local evaluation coordination 

■ Kmowledge-generating product 
development 


■ Evaluation plan outline 

■ Process and outcomes evaluation designs 

■ SDU, clinician, cost, and client measures 

■ Roles and responsibilities for grantee 
provider/evaluator staff 

■ Guidelines for evaluation products 


GRANTEES: 

MULTI-SITE 

EVALUATORS 


■ Grant applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 


NATIONAL 

EVALUATORS/ 

SERVICES 

RESEARCHERS 


■ Contract applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 



IEM products and other evaluation materials may be obtained from: 

http://neds.calib.com 
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Editor’s Note 



This document is one of a series of papers that describe CSAT’s approach to substance 
abuse treatment evaluation. The graphic below illustrates the continuous evaluation knowledge 
development and application process which characterizes CSAT’s approach. At the core is the 
self-adjusting treatment evaluation model which is the foundation. The model integrates 
continuous, state-of-the-art evaluation with planning, management, operation, and service 
delivery within a multi-disciplinary learning community. Implementation of this model requires 
building of team capabilities, appropriate, state-of-the-art performance evaluation and cost 
analysis, and assimilation of CSAT’s integrative approach to treatment evaluation and integrative 
methodologies. Each of these processes work together to ensure continuous improvement. 



Ensuring Continuous Evaluation Knowledge Development and 

Application 




Assimilate Approach to * Ensuring Continuous, State-of-the-Art p| an an( j Conduct 

Treatment Evaluation Evaluation Performance Evaluation 
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Substance abuse treatment providers are increasingly called upon to demonstrate that they 
are delivering appropriate services, that those services have the desired impact, and that the 
services justify the costs. An ongoing process of evaluation and systems/services improvement 
integrated into the day-to-day operation of treatment providers is needed to do so. In addition, 
the evaluation and improvement process requires a multi-disciplinary team that includes 
treatment personnel, evaluators, Federal and State agencies, advocacy groups, funding agencies, 
and the community. Building team capability is integral to this approach. Treatment staff must 
be involved in knowledge development and application (i.e., planning and implementing 
evaluation efforts, incorporating changes in response to new knowledge, and sharing of 
findings). 
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